Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AIR] Checkpoint improvements #30948

Merged
merged 14 commits into from
Dec 9, 2022
Merged

[AIR] Checkpoint improvements #30948

merged 14 commits into from
Dec 9, 2022

Conversation

Yard1
Copy link
Member

@Yard1 Yard1 commented Dec 7, 2022 •
edited
Loading

Why are these changes needed?

This PR makes the following improvements to Checkpoint:

  • We now ensure that the canonical path is stored as _local_path and that the comparison in to_directory between _local_path and user passed path compares the paths and not strings.
  • We avoid calling to_dict() twice in to_bytes()
  • A utility method is added to set checkpoint metadata.
  • After calling from_checkpoint, the metadata will be copied from the source checkpoint to the target checkpoint.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@Yard1
Update horovod, install after ml_docker
2094b5d 
Signed-off-by: Antoni Baum <[email protected]>
@Yard1
Add install-horovod
62164bb 
Signed-off-by: Antoni Baum <[email protected]>
@Yard1
WIP
5aa50fc 
Signed-off-by: Antoni Baum <[email protected]>
@Yard1
WIP
14b7794 
Signed-off-by: Antoni Baum <[email protected]>
krfricke
krfricke requested changes Dec 8, 2022
View reviewed changes
Copy link
Contributor

@krfricke krfricke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, minor change please

python/ray/air/checkpoint.py Outdated
local_path=other._local_path,
data_dict=other._data_dict,
uri=other._uri,
obj_ref=other._obj_ref,
)
other._copy_metadata_attrs_to(new_checkpoint)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's reverse this (copy metadata from)

@Yard1 Yard1 marked this pull request as ready for review December 8, 2022 03:06
bveeramani
bveeramani reviewed Dec 8, 2022
View reviewed changes
python/ray/air/checkpoint.py Outdated Show resolved Hide resolved
python/ray/air/checkpoint.py Outdated Show resolved Hide resolved
python/ray/air/checkpoint.py Outdated Show resolved Hide resolved
python/ray/air/checkpoint.py Show resolved Hide resolved
python/ray/air/checkpoint.py Show resolved Hide resolved
@Yard1 Yard1 requested a review from krfricke December 8, 2022 22:50
Yard1
Yard1 commented Dec 8, 2022
View reviewed changes
python/ray/air/checkpoint.py Outdated Show resolved Hide resolved
@Yard1 @bveeramani
Apply suggestions from code review
375d4fa 
Co-authored-by: Balaji Veeramani <[email protected]>
Signed-off-by: Antoni Baum <[email protected]>
@Yard1 Yard1 mentioned this pull request Dec 9, 2022
Merged
7 tasks
krfricke
krfricke approved these changes Dec 9, 2022
View reviewed changes
Copy link
Contributor

@krfricke krfricke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@krfricke krfricke merged commit 3a1bee2 into ray-project:master Dec 9, 2022
@Yard1 Yard1 deleted the air_checkpoint_improvements branch December 9, 2022 23:04
WeichenXu123 pushed a commit to WeichenXu123/ray that referenced this pull request Dec 19, 2022
@Yard1 @bveeramani @WeichenXu123
[AIR] Checkpoint improvements (ray-project#30948)
7a0985b 
Boston dataset (used in tests) is/will be removed from sklearn.

Signed-off-by: Antoni Baum <[email protected]>
Co-authored-by: Balaji Veeramani <[email protected]>
Signed-off-by: Weichen Xu <[email protected]>
tamohannes pushed a commit to ju2ez/ray that referenced this pull request Jan 25, 2023
@Yard1 @bveeramani @tamohannes
[AIR] Checkpoint improvements (ray-project#30948)
0a11946 
Boston dataset (used in tests) is/will be removed from sklearn.

Signed-off-by: Antoni Baum <[email protected]>
Co-authored-by: Balaji Veeramani <[email protected]>
Signed-off-by: tmynn <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bveeramani bveeramani bveeramani left review comments

@krfricke krfricke krfricke approved these changes

Assignees

@krfricke krfricke

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Yard1 @krfricke @bveeramani